Optic Flow
   HOME

TheInfoList



OR:

Optical flow or optic flow is the pattern of apparent
motion In physics, motion is the phenomenon in which an object changes its position with respect to time. Motion is mathematically described in terms of displacement, distance, velocity, acceleration, speed and frame of reference to an observer and mea ...
of objects, surfaces, and edges in a visual scene caused by the relative motion between an observer and a scene. Optical flow can also be defined as the distribution of apparent velocities of movement of brightness pattern in an image. The concept of optical flow was introduced by the American psychologist
James J. Gibson James Jerome Gibson (; January 27, 1904 – December 11, 1979) was an American psychologist and is considered to be one of the most important contributors to the field of visual perception. Gibson challenged the idea that the nervous system ...
in the 1940s to describe the visual stimulus provided to animals moving through the world. Gibson stressed the importance of optic flow for affordance perception, the ability to discern possibilities for action within the environment. Followers of Gibson and his ecological approach to psychology have further demonstrated the role of the optical flow stimulus for the perception of movement by the observer in the world; perception of the shape, distance and movement of objects in the world; and the control of
locomotion Locomotion means the act or ability of something to transport or move itself from place to place. Locomotion may refer to: Motion * Motion (physics) * Robot locomotion, of man-made devices By environment * Aquatic locomotion * Flight * Locomo ...
. The term optical flow is also used by roboticists, encompassing related techniques from image processing and control of navigation including motion detection, object segmentation, time-to-contact information, focus of expansion calculations, luminance, motion compensated encoding, and stereo disparity measurement.


Estimation

Sequences of ordered images allow the estimation of motion as either instantaneous image velocities or discrete image displacements. Fleet and Weiss provide a tutorial introduction to gradient based optical flow. John L. Barron, David J. Fleet, and Steven Beauchemin provide a performance analysis of a number of optical flow techniques. It emphasizes the accuracy and density of measurements. The optical flow methods try to calculate the motion between two image frames which are taken at times t and t+\Delta t at every
voxel In 3D computer graphics, a voxel represents a value on a regular grid in three-dimensional space. As with pixels in a 2D bitmap, voxels themselves do not typically have their position (i.e. coordinates) explicitly encoded with their values. Ins ...
position. These methods are called differential since they are based on local
Taylor series In mathematics, the Taylor series or Taylor expansion of a function is an infinite sum of terms that are expressed in terms of the function's derivatives at a single point. For most common functions, the function and the sum of its Taylor serie ...
approximations of the image signal; that is, they use partial derivatives with respect to the spatial and temporal coordinates. For a (2D + ''t'')-dimensional case (3D or ''n''-D cases are similar) a voxel at location (x,y,t) with intensity I(x,y,t) will have moved by \Delta x, \Delta y and \Delta t between the two image frames, and the following ''brightness constancy constraint'' can be given: :I(x,y,t) = I(x+\Delta x, y + \Delta y, t + \Delta t) Assuming the movement to be small, the image constraint at I(x,y,t) with
Taylor series In mathematics, the Taylor series or Taylor expansion of a function is an infinite sum of terms that are expressed in terms of the function's derivatives at a single point. For most common functions, the function and the sum of its Taylor serie ...
can be developed to get: :I(x+\Delta x,y+\Delta y,t+\Delta t) = I(x,y,t) + \frac\,\Delta x+\frac\,\Delta y+\frac \, \Delta t+
higher-order terms In mathematics and applied mathematics, perturbation theory comprises methods for finding an approximate solution to a problem, by starting from the exact solution of a related, simpler problem. A critical feature of the technique is a middle ...
By truncating the higher order terms (which performs a linearization) it follows that: :\frac\Delta x+\frac\Delta y+\frac\Delta t = 0 or, dividing by \Delta t, :\frac\frac + \frac\frac + \frac \frac = 0 which results in :\fracV_x+\fracV_y+\frac = 0 where V_x,V_y are the x and y components of the velocity or optical flow of I(x,y,t) and \tfrac, \tfrac and \tfrac are the derivatives of the image at (x,y,t) in the corresponding directions. I_x, I_y and I_t can be written for the derivatives in the following. Thus: :I_xV_x+I_yV_y=-I_t or :\nabla I\cdot\vec = -I_t This is an equation in two unknowns and cannot be solved as such. This is known as the ''
aperture problem Motion perception is the process of inferring the speed and direction of elements in a scene based on visual, vestibular and proprioceptive inputs. Although this process appears straightforward to most observers, it has proven to be a difficult pr ...
'' of the optical flow algorithms. To find the optical flow another set of equations is needed, given by some additional constraint. All optical flow methods introduce additional conditions for estimating the actual flow.


Methods for determination

*
Phase correlation Phase correlation is an approach to estimate the relative translative offset between two similar images (digital image correlation) or other data sets. It is commonly used in image registration and relies on a frequency-domain representation of t ...
– inverse of normalized cross-power spectrum *Block-based methods – minimizing sum of squared differences or
sum of absolute differences In digital image processing, the sum of absolute differences (SAD) is a measure of the similarity between image blocks. It is calculated by taking the absolute difference between each pixel in the original block and the corresponding pixel in the ...
, or maximizing normalized cross-correlation *Differential methods of estimating optical flow, based on partial derivatives of the image signal and/or the sought flow field and higher-order partial derivatives, such as: **
Lucas–Kanade method In computer vision, the Lucas–Kanade method is a widely used differential method for optical flow estimation developed by Bruce D. Lucas and Takeo Kanade. It assumes that the flow is essentially constant in a local neighbourhood of the pixel und ...
– regarding image patches and an affine model for the flow field **
Horn–Schunck method The Horn–Schunck method of estimating optical flow is a global method which introduces a global constraint of ''smoothness'' to solve the ''aperture problem'' (see Optical Flow for further description). Mathematical details The Horn-Schunck alg ...
– optimizing a functional based on residuals from the brightness constancy constraint, and a particular regularization term expressing the expected smoothness of the flow field ** Buxton–Buxton method – based on a model of the motion of edges in image sequences ** Black–Jepson method – coarse optical flow via correlation **General
variational methods The calculus of variations (or Variational Calculus) is a field of mathematical analysis that uses variations, which are small changes in functions and functionals, to find maxima and minima of functionals: mappings from a set of functions t ...
– a range of modifications/extensions of Horn–Schunck, using other data terms and other smoothness terms. *Discrete optimization methods – the search space is quantized, and then image matching is addressed through label assignment at every pixel, such that the corresponding deformation minimizes the distance between the source and the target image. The optimal solution is often recovered through
Max-flow min-cut theorem In computer science and optimization theory, the max-flow min-cut theorem states that in a flow network, the maximum amount of flow passing from the ''source'' to the ''sink'' is equal to the total weight of the edges in a minimum cut, i.e., the ...
algorithms, linear programming or
belief propagation A belief is an attitude that something is the case, or that some proposition is true. In epistemology, philosophers use the term "belief" to refer to attitudes about the world which can be either true or false. To believe something is to take i ...
methods. Many of these, in addition to the current state-of-the-art algorithms are evaluated on the Middlebury Benchmark Dataset. Other popular benchmark datasets are
KITTI These datasets are applied for machine learning research and have been cited in peer-reviewed academic journals. Datasets are an integral part of the field of machine learning. Major advances in this field can result from advances in learning a ...
and
Sintel ''Sintel'', code-named ''Project Durian'' during production, is a 2010 computer-animated fantasy short film. It was the third Blender "open movie". It was produced by Ton Roosendaal, chairman of the Blender Foundation, written by Esther Wouda ...
.


Uses

Motion estimation Motion estimation is the process of determining ''motion vectors'' that describe the transformation from one 2D image to another; usually from adjacent frames in a video sequence. It is an ill-posed problem as the motion is in three dimensions ...
and
video compression In information theory, data compression, source coding, or bit-rate reduction is the process of encoding information using fewer bits than the original representation. Any particular compression is either lossy or lossless. Lossless compression ...
have developed as a major aspect of optical flow research. While the optical flow field is superficially similar to a dense motion field derived from the techniques of motion estimation, optical flow is the study of not only the determination of the optical flow field itself, but also of its use in estimating the three-dimensional nature and structure of the scene, as well as the 3D motion of objects and the observer relative to the scene, most of them using the image Jacobian. Optical flow was used by robotics researchers in many areas such as:
object detection Object detection is a computer technology related to computer vision and image processing that deals with detecting instances of semantic objects of a certain class (such as humans, buildings, or cars) in digital images and videos. Well-researched ...
and tracking, image dominant plane extraction, movement detection, robot navigation and
visual odometry In robotics and computer vision, visual odometry is the process of determining the position and orientation of a robot by analyzing the associated camera images. It has been used in a wide variety of robotic applications, such as on the Mars Expl ...
. Optical flow information has been recognized as being useful for controlling micro air vehicles. The application of optical flow includes the problem of inferring not only the motion of the observer and objects in the scene, but also the
structure A structure is an arrangement and organization of interrelated elements in a material object or system, or the object or system so organized. Material structures include man-made objects such as buildings and machines and natural objects such as ...
of objects and the environment. Since awareness of motion and the generation of mental maps of the structure of our environment are critical components of animal (and human)
vision Vision, Visions, or The Vision may refer to: Perception Optical perception * Visual perception, the sense of sight * Visual system, the physical mechanism of eyesight * Computer vision, a field dealing with how computers can be made to gain un ...
, the conversion of this innate ability to a computer capability is similarly crucial in the field of
machine vision Machine vision (MV) is the technology and methods used to provide imaging-based automatic inspection and analysis for such applications as automatic inspection, process control, and robot guidance, usually in industry. Machine vision refers to ...
. Consider a five-frame clip of a ball moving from the bottom left of a field of vision, to the top right. Motion estimation techniques can determine that on a two dimensional plane the ball is moving up and to the right and vectors describing this motion can be extracted from the sequence of frames. For the purposes of video compression (e.g.,
MPEG The Moving Picture Experts Group (MPEG) is an alliance of working groups established jointly by International Organization for Standardization, ISO and International Electrotechnical Commission, IEC that sets standards for media coding, includ ...
), the sequence is now described as well as it needs to be. However, in the field of machine vision, the question of whether the ball is moving to the right or if the observer is moving to the left is unknowable yet critical information. Not even if a static, patterned background were present in the five frames, could we confidently state that the ball was moving to the right, because the pattern might have an infinite distance to the observer.


Optical flow sensor

An optical flow sensor is a vision sensor capable of measuring optical flow or visual motion and outputting a measurement based on optical flow. Various configurations of optical flow sensors exist. One configuration is an image sensor chip connected to a processor programmed to run an optical flow algorithm. Another configuration uses a vision chip, which is an integrated circuit having both the
image sensor An image sensor or imager is a sensor that detects and conveys information used to make an image. It does so by converting the variable attenuation of light waves (as they pass through or reflect off objects) into signals, small bursts of curr ...
and the processor on the same die, allowing for a compact implementation. An example of this is a generic optical mouse sensor used in an
optical mouse An optical mouse is a computer mouse which uses a light source, typically a light-emitting diode (LED), and a light detector, such as an array of photodiodes, to detect movement relative to a surface. Variations of the optical mouse have largely r ...
. In some cases the processing circuitry may be implemented using analog or mixed-signal circuits to enable fast optical flow computation using minimal current consumption. One area of contemporary research is the use of
neuromorphic engineering Neuromorphic engineering, also known as neuromorphic computing, is the use of electronic circuits to mimic neuro-biological architectures present in the nervous system. A neuromorphic computer/chip is any device that uses physical artificial ...
techniques to implement circuits that respond to optical flow, and thus may be appropriate for use in an optical flow sensor. Such circuits may draw inspiration from biological neural circuitry that similarly responds to optical flow. Optical flow sensors are used extensively in computer optical mice, as the main sensing component for measuring the motion of the mouse across a surface. Optical flow sensors are also being used in
robotics Robotics is an interdisciplinary branch of computer science and engineering. Robotics involves design, construction, operation, and use of robots. The goal of robotics is to design machines that can help and assist humans. Robotics integrat ...
applications, primarily where there is a need to measure visual motion or relative motion between the robot and other objects in the vicinity of the robot. The use of optical flow sensors in unmanned aerial vehicles (UAVs), for stability and obstacle avoidance, is also an area of current research.


See also

*
Ambient optic array The ambient optic array is the structured arrangement of light with respect to a point of observation. American psychologist James J. Gibson posited the existence of the ambient optic array as a central part of his ecological approach to optics. F ...
*
Optical mouse An optical mouse is a computer mouse which uses a light source, typically a light-emitting diode (LED), and a light detector, such as an array of photodiodes, to detect movement relative to a surface. Variations of the optical mouse have largely r ...
*
Range imaging Range imaging is the name for a collection of techniques that are used to produce a 2D image showing the distance to points in a scene from a specific point, normally associated with some type of sensor device. The resulting range image has pix ...
*
Vision processing unit A vision processing unit (VPU) is (as of 2018) an emerging class of microprocessor; it is a specific type of AI accelerator, designed to accelerate machine vision tasks. Overview Vision processing units are distinct from video processing units ...
*
Continuity Equation A continuity equation or transport equation is an equation that describes the transport of some quantity. It is particularly simple and powerful when applied to a conserved quantity, but it can be generalized to apply to any extensive quantity. S ...


References


External links


Finding Optic FlowArt of Optical Flow
article on fxguide.com (using optical flow in visual effects)
Optical flow evaluation and ground truth sequences.Middlebury Optical flow evaluation and ground truth sequences.mrf-registration.net
- Optical flow estimation through MRF
The French Aerospace Lab:
GPU implementation of a Lucas-Kanade based optical flow
CUDA Implementation
by CUVI (CUDA Vision & Imaging Library)
Horn and Schunck Optical Flow:
Online demo and source code of the Horn and Schunck method
TV-L1 Optical Flow:
Online demo and source code of the Zach et al. method
Robust Optical Flow:
Online demo and source code of the Brox et al. method {{DEFAULTSORT:Optical Flow Motion in computer vision